Display of a volume of captured video or positional tracking video may enable a viewer to perceive a captured scene from any location and at any viewing angle within a viewing volume. Using the data provided by such a video system, a viewpoint can be reconstructed to provide the view of a scene from any location within the viewing volume. When viewing this video with a virtual reality head-mounted display, the user may enjoy an immersive virtual presence within an environment. Such a virtual reality experience may be enhanced by providing viewer motion with six degrees of freedom, stereoscopic perception at any interpupillary distance, full motion parallax, and/or correct view-dependent lighting.
One key challenge to virtual reality (VR) and augmented reality (AR) video with full motion parallax and view-independent lighting is its immense data volume, which may be more than one hundred times larger than conventional 2D video. The large data requirement becomes prohibitive for viewers to store the content in consumer-grade devices, and also poses a challenge for distributors who wish to transmit the content over a network.
Virtual reality and augmented reality video may include depth cues such as stereopsis, binocular occlusions, vergence, motion parallax and view-dependent lighting, which may enhance the viewer's sense of immersion. Many existing lossy video compression techniques, such as chroma subsampling, transform coding, and quantization reduce the bit-rate of the video stream by discarding imperceptible information. However, such known techniques generally do not generally exploit additional aspects of the virtual reality or augmented reality video stream, and therefore do not provide compression ratios sufficient for use with virtual reality or augmented reality video with full motion parallax and view-dependent lighting.